Minimum Excess Risk in Bayesian Learning

We analyze the best achievable performance of Bayesian learning under generative models by defining and upper-bounding minimum excess risk (MER): gap between expected loss attainable from data that could be achieved if model realization were known. The definition MER provides a principled way to define different notions uncertainties in learning, including aleatoric uncertainty epistemic uncertainty. Two methods for deriving upper bounds are presented. first method, generally suitable with parametric model, upper-bounds conditional mutual information parameters quantity being predicted given observed data. It allows us quantify rate at which decays zero as more becomes available. Under realizable models, this method also relates richness function class, notably VC dimension binary classification. second particularly predictive estimation error via various continuity arguments. extend analysis setting multiple families nonparametric models. Along discussions we draw some comparisons frequentist learning.